This notebook reports code and results of the analysis conducted on the two versions of the JfreeChart software system included in the dataset (i.e. $0.6.0$ and $0.7.1$).
The goal of the analysis is to provide insights on the differences and/or similarities between the methods included in both systems, along with some considerations on the coherence of methods included only in one of the two.
Utilities functions used throughout this notebook. Feel free to skip to the **Analisys Section** directly.
In [1]:
# %load preamble_directives.py
"""Some imports and path settings to make notebook code
running smoothly.
"""
# Author: Valerio Maggio <valeriomaggio@gmail.com>
# Copyright (c) 2015 Valerio Maggio <valeriomaggio@gmail.com>
# License: BSD 3 clause
import sys, os
# Extending PYTHONPATH to allow relative import!
sys.path.append(os.path.join(os.path.abspath(os.path.curdir), '..'))
# Import Django Settings
from django.conf import settings
# Import Comments_Classification (Django) Project Settings
from comments_classification import settings as comments_classification_settings
try:
settings.configure(**comments_classification_settings.__dict__)
except RuntimeError:
# settings already configured
pass
# ---------------------
# Module Import Section
# ---------------------
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
In [2]:
import re
def strip_tags(text):
"""Strips all HTML tags from text"""
HTML_TAG_RE = re.compile(r'<[^>]+>')
return HTML_TAG_RE.sub('', text)
In [3]:
from string import punctuation
def strip_punctuations(text, allowed='@'):
"""Strips all the punctuation character from the input comment.
Parameters
----------
text : str
The input text to process
allowed : str, optional.
The list of punctuation characters to exclude from the processing
(default is '@' as for JavaDoc Comments).
"""
black_list = punctuation
for c in allowed:
black_list = black_list.replace(c, '')
for p in black_list:
text = text.replace(p, ' ')
return '\n'.join(' '.join(w for w in line.split() if len(w)) for line in text.splitlines() if len(line))
In [4]:
def normalise_lines(text):
"""Removes additional (trailing) spaces from lines of the given text and
returns it normalised (no extra spaces)"""
return '\n'.join(l.strip() for l in text.splitlines() if len(l.strip()))
In [5]:
def extract_locs(code_fragment):
"""Normalise and returns the lines of code in the input fragment."""
locs = ' '.join(l.strip() for st in code_fragment.split(';') for l in st.splitlines()
if len(l.strip()) and len(strip_punctuations(l.strip())))
return locs
In [6]:
class Pipeline(object):
"""Implements a simple linear pipeline process"""
def __init__(self, *callables):
"""Creates a new Pipeline of processes (i.e. `callables`.)
Each of this callable, must always return a value as it
will represent the new `data` parameter passed through
the pipeline till finally returned.
Parameters
----------
callables : list
A list of callables (e.g. functions) of arity one.
This list constitutes the set of filters of the pipeline.
"""
self._filters = list(callables)
@property
def filters(self):
return self._filters
def __iadd__(self, process):
if not callable(process):
raise ValueError('The value should be a callable')
from inspect import signature
sig_process = signature(process)
if len(sig_process.parameters) != 1:
raise ValueError('The input function must have arity one!')
self._filters.append(process)
return self
def process(self, data):
"""Execute the pipeline"""
for callable in self._filters:
data = callable(data)
return data
In [7]:
NOT_EVALUATED = -1
DONT_KNOW = 2
FURTHER_EVAL = 5
AGREEMENT = 3
STRONG_AGREEMENT = 4
def is_coherent(method):
"""Return wheter or not comment is coherent with its method implementation according to judges evaluations.
Parameters
----------
method : `source_code_analysis.models.CodeMethod`
Instance of a `CodeMethod` model holding the reference to its corresponding evaluations.
Returns
-------
bool :
True if the evaluation (the first one retrieved from the db) corresponds to an
AGREEMENT | STRONG_AGREEMENT value.
"""
return (method.agreement_evaluations.last().agreement_vote in (AGREEMENT, STRONG_AGREEMENT))
def has_agreement_evaluations(method):
"""Check that input methods has agreement evaluations interesting for the current analysis
(i.e. different from DONT_KNOW).
Parameters
----------
method : `source_code_analysis.models.CodeMethod`
Instance of a `CodeMethod` model holding the reference to its corresponding evaluations.
Returns
-------
bool :
True if the evaluation (the first one retrieved from the db) does **not**
correspond to a DONT_KNOW value.
"""
return (method.agreement_evaluations.last().agreement_vote not in (NOT_EVALUATED,
DONT_KNOW, FURTHER_EVAL))
In [8]:
def signature(code_fragment):
"""Returns the signature of a method extracted from the input code fragment.
Parameters
----------
code_fragment : str
The implementation code of a method (i.e. method.code_fragment attribute)
Returns
-------
str :
The signature string of the method
"""
first_line = code_fragment[:code_fragment.find('{')]
return ' '.join([l.strip() for l in first_line.splitlines() if len(l.strip())])
In [9]:
def gather_all_methods(sw_project):
"""Gather all methods for the input software project
Parameters
----------
sw_project : `source_code_analysis.models.SoftwareProject`
Target Software Project
Returns
-------
dict :
A dictionary mapping all methods with its unique key.
This key is extremely important to correctly identify similarities among
multiple versions of the same software.
In more details, the key for a single *method* is defined by the following triple:
* Name of the Source File
* Name of the Class
* Signature of the method
"""
# gather all methods
methods = filter(has_agreement_evaluations, sw_project.code_methods.all())
# create the map
methods_map = dict()
for method in methods:
key = '{}{}{}'.format(method.code_class.src_filename,
method.code_class.class_name,
signature(method.code_fragment))
if not key in methods_map:
methods_map[key] = method
else:
print('Key already present: ')
print('Current method ID: ', method.id)
print('Already present method: ', methods_map[key].id)
# return map
return methods_map
In [10]:
def randomly_pick_a_method_from(list_of_methods_keys, with_code=False, with_coherence=False):
"""Randomly pick a method from the input collection of keys and print the
corresponding lead comments. If `with_code` parameter is provided,
the code_fragment is printed as well.
Moreover, if the `with_coherence` parameter is True, the corresponding coherence evaluation is
reported in the output, as well.
"""
from random import choice
random_key = choice(list_of_methods_keys)
method_in_jf060 = jf060_methods[random_key]
method_in_jf071 = jf071_methods[random_key]
print('='*80)
print('Method in JFreeChart 0.6.0', end=' ')
if with_coherence:
print('Is Coherent: ', is_coherent(method_in_jf060))
else:
print('')
print(method_in_jf060.comment)
if with_code:
print('')
print(method_in_jf060.code_fragment)
print('\n\n')
print('Method in JFreeChart 0.7.1', end=' ')
if with_coherence:
print('Is Coherent: ', is_coherent(method_in_jf071))
else:
print('')
print(method_in_jf071.comment)
if with_code:
print('')
print(method_in_jf071.code_fragment)
print('='*80, end='\n\n')
In [11]:
from source_code_analysis.models import SoftwareProject
In [12]:
jfreechart_060 = SoftwareProject.objects.get(name__iexact='JFreeChart', version='0.6.0')
jfreechart_071 = SoftwareProject.objects.get(name__iexact='JFreeChart', version='0.7.1')
In [13]:
jf060_methods = gather_all_methods(jfreechart_060)
jf071_methods = gather_all_methods(jfreechart_071)
In [14]:
print('Total No. of Methods in JFreeChart 0.6.0: ', len(jf060_methods))
print('Total No. of Methods in JFreeChart 0.7.1: ', len(jf071_methods))
In [15]:
# Set of all the Keys in common between the two considered versions
methods_in_common = set(jf060_methods.keys()).intersection(set(jf071_methods.keys()))
print('Total Methods in Common: ', len(methods_in_common))
In [16]:
# Set the Pipelines
comment_pipeline = Pipeline(strip_tags, strip_punctuations, normalise_lines)
# List to store the references to methods
# sharing (or not) the same comment between the two versions
same_comment = list()
different_comment = list()
for mkey in methods_in_common:
comment_in_060 = comment_pipeline.process(jf060_methods[mkey].comment)
comment_in_071 = comment_pipeline.process(jf071_methods[mkey].comment)
if comment_in_060 == comment_in_071:
same_comment.append(mkey)
else:
different_comment.append(mkey)
In [17]:
print('Total Number of Methods in Common: ', len(methods_in_common))
print('\t No. of those Sharing the Same Comment: ', len(same_comment))
print('\t No. of those With Differences in Comment: ', len(different_comment))
In [18]:
# Test: get a random key and check that they actually share the same comments
# regardless the formattings (e.g. trailing spaces)
randomly_pick_a_method_from(same_comment) # Test 1
randomly_pick_a_method_from(same_comment) # Test 2
randomly_pick_a_method_from(same_comment) # Test 3
randomly_pick_a_method_from(same_comment) # Test 4
From the total set of $283$ methods in common between JFreeChart 0.6.0 and 0.7.1,
In [19]:
# Qualitative Analysis (preliminary)
randomly_pick_a_method_from(different_comment) # Test 1
randomly_pick_a_method_from(different_comment) # Test 2
randomly_pick_a_method_from(different_comment) # Test 3
randomly_pick_a_method_from(different_comment) # Test 4
In [20]:
methods_with_same_comments = same_comment # code readability purposes
# Set the Pipeline
code_pipeline = Pipeline(extract_locs)
same_comment_and_code = list()
same_comment_different_code = list()
for mkey in methods_with_same_comments:
code_in_060 = code_pipeline.process(jf060_methods[mkey].code_fragment)
code_in_071 = code_pipeline.process(jf071_methods[mkey].code_fragment)
if code_in_060 == code_in_071:
same_comment_and_code.append(mkey)
else:
same_comment_different_code.append(mkey)
In [21]:
print('Total Number of Common Methods with the same comments: ', len(methods_with_same_comments))
print('\t No. of those Sharing the Same Code: ', len(same_comment_and_code))
print('\t No. of those With Differences in Code: ', len(same_comment_different_code))
In [22]:
methods_with_different_comments = different_comment # code readability purposes
# Set the Pipeline
code_pipeline = Pipeline(extract_locs)
different_comment_same_code = list()
different_comment_and_code = list()
for mkey in methods_with_different_comments:
code_in_060 = code_pipeline.process(jf060_methods[mkey].code_fragment)
code_in_071 = code_pipeline.process(jf071_methods[mkey].code_fragment)
if code_in_060 == code_in_071:
different_comment_same_code.append(mkey)
else:
different_comment_and_code.append(mkey)
In [23]:
print('Total Number of Common Methods with different comments: ', len(methods_with_different_comments))
print('\t No. of those Sharing the Same Code: ', len(different_comment_same_code))
print('\t No. of those With Differences in Code: ', len(different_comment_and_code))
In [24]:
same_coherence = list()
coherence_changed = list()
for mkey in same_comment_and_code:
mth_in_060 = jf060_methods[mkey]
mth_in_071 = jf071_methods[mkey]
if is_coherent(mth_in_060) == is_coherent(mth_in_071):
same_coherence.append(mkey)
else:
coherence_changed.append(mkey)
In [25]:
print('Total Number of Methods Sharing the same Lead Comment and Code')
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))
In [26]:
same_coherence = list()
coherence_changed = list()
for mkey in same_comment_different_code:
mth_in_060 = jf060_methods[mkey]
mth_in_071 = jf071_methods[mkey]
if is_coherent(mth_in_060) == is_coherent(mth_in_071):
same_coherence.append(mkey)
else:
coherence_changed.append(mkey)
In [27]:
print('Total Number of Methods sharing the same lead comment but have different code', len(same_comment_different_code))
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))
In [28]:
# Try to spot some insights
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True) # Method 1
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True) # Method 2
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True) # Method 3
randomly_pick_a_method_from(same_comment_different_code, with_code=True, with_coherence=True) # Method 4
Among those methods sharing the same lead comments but have differences in the implementations ($32$ in total), None of them have differences in the coherence evaluation. This means that differences in the implementation were limited to syntactic constructs and names of the variables.
In [29]:
same_coherence = list()
coherence_changed = list()
for mkey in different_comment_same_code:
mth_in_060 = jf060_methods[mkey]
mth_in_071 = jf071_methods[mkey]
if is_coherent(mth_in_060) == is_coherent(mth_in_071):
same_coherence.append(mkey)
else:
coherence_changed.append(mkey)
In [30]:
print('Total Number of Methods having different comment but the same code', len(different_comment_same_code))
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))
In [31]:
# Try to slpot some insights
randomly_pick_a_method_from(coherence_changed, with_code=True, with_coherence=True) # Method 1
randomly_pick_a_method_from(coherence_changed, with_code=True, with_coherence=True) # Method 2
This is interesting.
There are just 2 cases in which differences in comments (and not in implementation) lead to different coherence evaluations.
As a matter of facts, the changes in the lead comments for methods gathered from JFreeChart 0.7.1 are not compliant
with the corresponding code. For instance, in the latter case,
the parameters listed in the Javadoc @param annotations are not all compliant with the corresponding method signature.
This phenomenon may be likely due to refactoring changes (i.e. renaming of variables) not reflected in the comment.
In [32]:
# Try to spot some insights
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 1
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 2
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 3
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 4
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 5
This is also interesting.
In all other cases of common methods having differences in the lead comment but not in the implementation, it seems that the corresponding coherence is not affected.
As a matter of facts, all the changes and differences in the lead comments for methods gathered from JFreeChart 0.7.1 are limited to Javadoc syntax adjustments, typos corrections, and revisions of parameters and method's descriptions.
In [33]:
same_coherence = list()
coherence_changed = list()
for mkey in different_comment_and_code:
mth_in_060 = jf060_methods[mkey]
mth_in_071 = jf071_methods[mkey]
if is_coherent(mth_in_060) == is_coherent(mth_in_071):
same_coherence.append(mkey)
else:
coherence_changed.append(mkey)
In [34]:
print('Total Number of Methods having different comment and code', len(different_comment_and_code))
print('\t Same Coherence: ', len(same_coherence))
print('\t Different Coherence', len(coherence_changed))
In [35]:
# Try to spot some insights
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 1
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 2
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 3
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 4
randomly_pick_a_method_from(same_coherence, with_code=True, with_coherence=True) # Method 5
This is interesting as well!!.
Among the $10$ methods in common between the two versions that have differences in code and comments, $9$ of them have the same evaluation of coherence.
This phenomenon reflects the fact that in these $9$ cases, code and comments have been updated accordingly!
However, if we look at the totality of common methods, this number is ridiculous!
In more details, it is $9$ methods out of $58$, $\approx 16\%$, where $58$ corresponds to $283$ (methods in common) minus those with no real difference in code and comments ($225$ - the majority of them)!!
In [36]:
# Try to spot some insights
randomly_pick_a_method_from(coherence_changed, with_code=True, with_coherence=True) # Method 1
This is interesting (and correct)!!.
In the only case (out of 10) where there is difference in the coherence evaluation between the two methods in common, the corresponding lead comments and implementations have been updated accordingly!
In fact, while there was no coherence for the method extracted from JFreeChart 0.6.0, there is for the one from JfreeChart 0.7.1.
In [37]:
methods_in_060 = set(jf060_methods.keys())
methods_in_071 = set(jf071_methods.keys())
# Set of Methods in 0.6.0 and not in 0.7.1
methods_in_060_not_in_071 = methods_in_060.difference(methods_in_071)
print('Total No. of Methods in JfreeChart 0.6.0 and NOT in 0.7.1: ', len(methods_in_060_not_in_071))
print('-'*80)
# Set of Methods in 0.7.1 and not in 0.6.0
methods_in_071_not_in_060 = methods_in_071.difference(methods_in_060)
print('Total No. of Methods in JfreeChart 0.7.1 and NOT in 0.6.0: ', len(methods_in_071_not_in_060))
In [38]:
coherent = list()
not_coherent = list()
for mkey in methods_in_060_not_in_071:
method = jf060_methods[mkey]
if is_coherent(method):
coherent.append(mkey)
else:
not_coherent.append(mkey)
print('Total Number of Methods in JFreeChart 0.6.0 but NOT in 0.7.1:', len(methods_in_060_not_in_071))
print('\t Coherent: ', len(coherent))
print('\t Not Coherent', len(not_coherent))
In [39]:
coherent = list()
not_coherent = list()
for mkey in methods_in_071_not_in_060:
method = jf071_methods[mkey]
if is_coherent(method):
coherent.append(mkey)
else:
not_coherent.append(mkey)
print('Total Number of Methods in JFreeChart 0.7.1 but NOT in 0.6.0:', len(methods_in_071_not_in_060))
print('\t Coherent: ', len(coherent))
print('\t Not Coherent', len(not_coherent))
In this section, the main purpose of the analysis is to try to check some possible matches between methods not in JFreeChart 0.6.0 but present in JFreeChart 0.7.1.
The main idea is that we would like to find (or guess) all those methods whose signature has been changed/updated thus not appearing in the set of Common Methods.
In [40]:
# Analyse the differences: Try to guess if there is some method that has been CHANGED between the two versions
from collections import defaultdict
associations_map = defaultdict(list)
def match(method_key, target_class, target_file, target_signature_stub):
"""We try to infer a possible matching between two methods if they share:
- the same class name
- the same source file name
- their signature starts with the same _stub_
In particular, the `target_signature_stub` corresponds to the
first part of the signature till the first open paranthesis, i.e. "("
"""
mth = jf060_methods[method_key]
return (method_key not in jf071_methods and
mth.code_class.src_filename == target_file
and mth.code_class.class_name == target_class and
signature(mth.code_fragment).startswith(target_signature_stub))
# Iterate over all methods in JFreeChart 0.7.1 and NOT in 0.6.0
# in order to guess some possible signature matchings
for mkey in methods_in_071_not_in_060:
method = jf071_methods[mkey]
signature_071 = signature(method.code_fragment)
# get all the methods whose signature starts similarly to the target method
signature_stub = signature_071[:signature_071.find('(')]
class_name = method.code_class.class_name
src_file = method.code_class.src_filename
associations = list(filter(lambda k: match(k, class_name, src_file, signature_stub),
jf060_methods.keys()))
associations_map[mkey] = associations
In [41]:
# Filter out all that had no matching in the first place
possible_mappings = {k:v for k, v in associations_map.items() if len(v)}
In [42]:
print('We inferred a total of {} matchings for {} methods NOT in common!'.format(len(possible_mappings),
len(methods_in_071_not_in_060)))
In [43]:
# Print Matchings Guessed
for mkey, associations in possible_mappings.items():
print('Target Signature: ', signature(jf071_methods[mkey].code_fragment))
print('Possible Associations in Total: ', len(associations))
for i, assoc in enumerate(associations):
print('\t {}): '.format(i+1), assoc, end="\n\n")
print('-'*80)
Most of the guessed matchings are reasonable - most of them have just one possible match. Of course, we should manually dive more deeply into the code to check whether or not these methods actually corresponds to the same methods that has been changed between the two versions.
However, we may conclude that most of the methods present in the JFreeChart 0.7.1 version (but not in version 0.6.0) have been added in the new version!
JFreeChart 0.6.0 and 0.7.1 have $283$ methods in common
(Name of the File, Name of the Class, Method Signature).Methods in Common have been further analysed (and grouped) in terms of similarities (differences) in their lead comment and implementation. In more details:
As for the Comment:
As for the Implementation:
$32$ methods sharing the same comment have different implementation.
Similarly:
As for the Coherence:
For methods with the same comment but different code, we have
try catch blocks addition) which did not affect the semantic of the whole implementation.For methods with different comment but the same code, we have:
For methods with different code and comment, we have: